Development of a SGM-based multi-view reconstruction framework for aerial imagery

Advances in the technology of digital airborne camera systems allow for the observation of surfaces with sampling rates in the range of a few centimeters. In combination with novel matching approaches, which estimate depth information for virtually every pixel, surface reconstructions of impressive density and precision can be generated. Therefore, image based surface generation meanwhile is a serious alternative to LiDAR based data collection for many applications. Surface models serve as primary base for geographic products as for example map creation, production of true-ortho photos or visualization purposes within the framework of virtual globes. The goal of the presented theses is the development of a framework for the fully automatic generation of 3D surface models based on aerial images - both standard nadir as well as oblique views. This comprises several challenges. On the one hand dimensions of aerial imagery is considerable and the extend of the areas to be reconstructed can encompass whole countries. Beside scalability of methods this also requires decent processing times and efficient handling of the given hardware resources. Moreover, beside high precision requirements, a high degree of automation has to be guaranteed to limit manual interaction as much as possible. Due to the advantages of scalability, a stereo method is utilized in the presented thesis. The approach for dense stereo is based on an adapted version of the semi global matching (SGM) algorithm. Following a hierarchical approach corresponding image regions and meaningful disparity search ranges are identified. It will be verified that, dependent on undulations of the scene, time and memory demands can be reduced significantly, by up to 90% within some of the conducted tests. This enables the processing of aerial datasets on standard desktop machines in reasonable times even for large fields of depth. Stereo approaches generate disparity or depth maps, in which redundant depth information is available. To exploit this redundancy, a method for the refinement of stereo correspondences is proposed. Thereby redundant observations across stereo models are identified, checked for geometric consistency and their reprojection error is minimized. This way outliers are removed and precision of depth estimates is improved. In order to generate consistent surfaces, two algorithms for depth map fusion were developed. The first fusion strategy aims for the generation of 2.5D height models, also known as digital surface models (DSM). The proposed method improves existing methods regarding quality in areas of depth discontinuities, for example at roof edges. Utilizing benchmarks designed for the evaluation of image based DSM generation we show that the developed approaches favorably compare to state-of-the-art algorithms and that height precisions of few GSDs can be achieved. Furthermore, methods for the derivation of meshes based on DSM data are discussed. The fusion of depth maps for 3D scenes, as e.g. frequently required during evaluation of high resolution oblique aerial images in complex urban environments, demands for a different approach since scenes can in general not be represented as height fields. Moreover, depths across depth maps possess varying precision and sampling rates due to variances in image scale, errors in orientation and other effects. Within this thesis a median-based fusion methodology is proposed. By using geometry-adaptive triangulation of depth maps depth-wise normals are extracted and, along the point coordinates are filtered and fused using tree structures. The output of this method are oriented points which then can be used to generate meshes. Precision and density of the method will be evaluated using established multi-view benchmarks. Beside the capability to process close range datasets, results for large oblique airborne data sets will be presented. The report closes with a summary, discussion of limitations and perspectives regarding improvements and enhancements. The implemented algorithms are core elements of the commercial software package SURE, which is freely available for scientific purposes

Semantically Informed Multiview Surface Refinement

We present a method to jointly refine the geometry and semantic segmentation of 3D surface meshes. Our method alternates between updating the shape and the semantic labels. In the geometry refinement step, the mesh is deformed with variational energy minimization, such that it simultaneously maximizes photo-consistency and the compatibility of the semantic segmentations across a set of calibrated images. Label-specific shape priors account for interactions between the geometry and the semantic labels in 3D. In the semantic segmentation step, the labels on the mesh are updated with MRF inference, such that they are compatible with the semantic segmentations in the input images. Also, this step includes prior assumptions about the surface shape of different semantic classes. The priors induce a tight coupling, where semantic information influences the shape update and vice versa. Specifically, we introduce priors that favor (i) adaptive smoothing, depending on the class label; (ii) straightness of class boundaries; and (iii) semantic labels that are consistent with the surface orientation. The novel mesh-based reconstruction is evaluated in a series of experiments with real and synthetic data. We compare both to state-of-the-art, voxel-based semantic 3D reconstruction, and to purely geometric mesh refinement, and demonstrate that the proposed scheme yields improved 3D geometry as well as an improved semantic segmentation

Fast and robust generation of semantic urban terrain models from UAV video streams

We present an algorithm for extracting Level of Detail 2 (LOD2) building models from video streams captured by Unmaned Aerial Vehicles (UAVs). Typically, such imagery is of limited radiometric quality but the surface is captured with large redundancy. The first contribution of this paper is a novel algorithm exploiting this redundancy for precise depth computation. This is realized by fusing consistent depth estimations across single stereo models and generating a 2.5D elevation map from the resulting point clouds. Disparity maps are derived by a coarse-to-fine Semi-Global-Matching (SGM) method performing well on noisy imagery. The second contribution concerns a challenging step of the context-based urban terrain modeling: Dominant planes extraction for building reconstruction. Because of noisy data and complicated roof structures, both dominant plane parameters and initial values for support sets of planes are obtained by the J-Linkage algorithm. An improved pointto-plane labeling is presented to encourage the assignment of proximate points to the same plane. This is accomplished by non-local, Markov Random Field (MRF) - based optimization and segmentation of color information. The potential and the limitations of the proposed methods are shown using an UAV video sequence of limited radiometric quality

Lake ice monitoring with webcams

Continuous monitoring of climate indicators is important for understanding the dynamics and trends of the climate system. Lake ice has been identified as one such indicator, and has been included in the list of Essential Climate Variables (ECVs). Currently there are two main ways to survey lake ice cover and its change over time, in-situ measurements and satellite remote sensing. The challenge with both of them is to ensure sufficient spatial and temporal resolution. Here, we investigate the possibility to monitor lake ice with video streams acquired by publicly available webcams. Main advantages of webcams are their high temporal frequency and dense spatial sampling. By contrast, they have low spectral resolution and limited image quality. Moreover, the uncontrolled radiometry and low, oblique viewpoints result in heavily varying appearance of water, ice and snow. We present a workflow for pixel-wise semantic segmentation of images into these classes, based on state-of-the-art encoder-decoder Convolutional Neural Networks (CNNs). The proposed segmentation pipeline is evaluated on two sequences featuring different ground sampling distances. The experiment suggests that (networks of) webcams have great potential for lake ice monitoring. The overall per-pixel accuracies for both tested data sets exceed 95 %. Furthermore, per-image discrimination between ice-on and ice-off conditions, derived by accumulating per-pixel results, is 100 % correct for our test data, making it possible to precisely recover freezing and thawing dates.ISSN:2194-9042ISSN:2194-905

Lake Ice Monitoring with Webcams and Crowd-Sourced Images

© 2020 Copernicus GmbH. All rights reserved. Lake ice is a strong climate indicator and has been recognised as part of the Essential Climate Variables (ECV) by the Global Climate Observing System (GCOS). The dynamics of freezing and thawing, and possible shifts of freezing patterns over time, can help in understanding the local and global climate systems. One way to acquire the spatiooral information about lake ice formation, independent of clouds, is to analyse webcam images. This paper intends to move towards a universal model for monitoring lake ice with freely available webcam data. We demonstrate good performance, including the ability to generalise across different winters and lakes, with a state-of-the-art Convolutional Neural Network (CNN) model for semantic image segmentation, Deeplab v3+. Moreover, we design a variant of that model, termed Deep-U-Lab, which predicts sharper, more correct segmentation boundaries. We have tested the model's ability to generalise with data from multiple camera views and two different winters. On average, it achieves Intersection-over-Union (IoU) values of ≈71% across different cameras and ≈69% across different winters, greatly outperforming prior work. Going even further, we show that the model even achieves 60% IoU on arbitrary images scraped from photo-sharing websites. As part of the work, we introduce a new benchmark dataset of webcam images, Photi-LakeIce, from multiple cameras and two different winters, along with pixel-wise ground truth annotations.ISSN:2194-9042ISSN:2194-905

Exact convex formulations of network-oriented optimal operator placement

Abstract—Data processing tasks are increasingly spread across the internet to account for the spatially distributed nature of many data sources. In order to use network resources efficiently, subtasks need to be distributed in the network so data can be filtered close to the data sources. Previous approaches to this operator placement problem relied on various heuristics to constrain the complexity of the problem. In this paper, we propose two generic integer constrained problem formulations: a topology aware version which provides a placement including the specific network links as well as an end-to-end delay aware version which relies on the routing capabilities of the network. A linear programming relaxation for both versions is provided which allows exact and efficient solution using common solvers. I

Cloud-based 3D Reconstruction of Cultural Heritage Monuments using Open Access Image Repositories

A large number of photographs of cultural heritage items and monuments is publicly available in various Open Access Image Repositories (OAIR) and social media sites. Metadata inserted by camera, user and host site may help to determine the photograph content, geo-location and date of capture, thus allowing us, with relative success, to localise photos in space and time. Additionally, developments in Photogrammetry and Computer Vision, such as Structure from Motion (SfM), provide a simple and cost-effective method of generating relatively accurate camera orientations and sparse and dense 3D point clouds from 2D images. Our main goal is to provide a software tool able to run on desktop or cluster computers or as a back end of a cloud-based service, enabling historians, architects, archaeologists and the general public to search, download and reconstruct 3D point clouds of historical monuments from hundreds of images from the web in a cost-effective manner. The end products can be further enriched with metadata and published. This paper describes a workflow for searching and retrieving photographs of historical monuments from OAIR, such as Flickr and Picasa, and using them to build dense point clouds using SfM and dense image matching techniques. Computational efficiency is improved by a technique which reduces image matching time by using an image connectivity prior derived from low-resolution versions of the original images. Benchmarks for two large datasets showing the respective efficiency gains are presented